OracleService #326

Tommo-L · 2020-08-18T13:01:35Z

No description provided.

neo-modules.sln

src/Oracle/Oracle.csproj

src/RpcServer/RpcServer.csproj

shargon · 2020-08-26T10:44:34Z

src/Oracle/Protocol/OracleHttpProtocol.cs

+                return input;
+
+            JObject beforeObject = JObject.Parse(input);
+            JArray afterObjects = new JArray(beforeObject.SelectTokens(filterArgs).ToArray()); // TODO 


Filter will be a json filter (XPATH), or a contract execution?

-> Contract execution it's harder because the execution fee but it's transparent to the nodes, if we use a XPATH filters we will have a lot of underterminism betwen different implementations or libraries.

JSONPath?

According to neo-project/neo#1259 it seems that was decided JSONPath but for me it has two important dissadvantages.

Cost of gas, we will not have control over how much time was spent in this filter.

Same implementation throughout the different systems.

But it's ok for me, I am not against to use JsonPath.

src/Oracle/Protocol/OracleHttpProtocol.cs

shargon · 2020-08-26T10:48:07Z

src/Oracle/Oracle.cs

+        private CancellationTokenSource cancelSource;
+
+        private static readonly IOracleProtocol Https = new OracleHttpProtocol();
+        private static readonly TimeSpan MaxTaskTimeout = TimeSpan.FromMinutes(5);


It should be configurable or stored in the policy

Should this be related to the fees?

src/Oracle/Oracle.cs

shargon · 2020-08-26T10:54:51Z

src/Oracle/Oracle.cs

+                {
+                    try
+                    {
+                        var url = new Uri("http://" + node + "/");


Maybe we should force oracles to have https with his publicKey

src/Oracle/Oracle.cs

shargon · 2020-08-26T11:01:35Z

src/Oracle/Oracle.cs

+            ECPoint[] nodes = NativeContract.Oracle.GetOracleNodes(snapshot);
+            if (task.Signs.Count >= nodes.Length / 3 && task.Tx != null)
+            {
+                var contract = Contract.CreateMultiSigContract(nodes.Length - (nodes.Length - 1) / 3, nodes);


Different than https://github.com/neo-project/neo-modules/pull/326/files#diff-cb9c5c742827ddb7719d684b62664515R200

shargon · 2020-08-26T11:04:17Z

We should add in rpc allowed nodes, or allowed client certificates, in order to allow or not only the oracle nodes, because otherwise anyone can fill the rpc queue and deny the service.
Oracle nodes should allow only oracles nodes to his rpc servers.

erikzhang · 2020-08-26T11:27:50Z

We should add in rpc allowed nodes, or allowed client certificates

I think it has been done in:

neo-modules/src/RpcServer/RpcServer/config.json

Line 7 in 83b3a06

"TrustedAuthorities": [],

Tommo-L · 2020-08-26T15:06:02Z

I'll fix them tomorrow.

…into async-oracle

src/Oracle/Oracle.cs

…into async-oracle

Tommo-L · 2021-01-12T02:45:28Z

@Tommo-L it seems that the fee is different than expected.

Fixed.

It's because the script of native contract has been adjusted. See https://github.com/neo-project/neo/blob/8b1509f3a2fb7fa62a958a997fef18b7a30cfe1a/src/neo/SmartContract/Native/NativeContract.cs#L41-L49

src/OracleService/OracleService.cs

cloud8little · 2021-01-14T08:57:43Z

Test passed with following scenarios:

http request, the request transaction and response transaction both on-chain. but the response TX executed with FAULT result.

https service is down, response code is Timeout.
getapplicationlog of the request TX, the oracle contract address get the GAS user paid.
After dealing with a round of oracle request and response, the oracle node get the reward 0.5 GAS.
Filter like "$.result.subdir" , null, empty string "" are all ok.
When response body is too large, return error

gasForResponse<Oracle.MinimumResponseFee

Consensus of Oracle nodes. [Updates]
Multiple oracle nodes work well.

@shargon @Tommo-L @erikzhang please help review the cases, in case I miss something.

Tommo-L · 2021-01-14T09:46:22Z

These cases are fine. Thanks.

erikzhang · 2021-01-14T09:48:25Z

Conflicts

Tommo-L · 2021-01-14T10:24:39Z

Conflicts

Done.

src/OracleService/OracleService.cs

cloud8little · 2021-01-15T02:03:16Z

Let me do a smoke test.

cloud8little · 2021-01-15T02:12:58Z

tests/Neo.Plugins.OracleService.Tests/UT_OracleService.cs

+            Assert.AreEqual(@"[""Acme Co""]", Utility.StrictUTF8.GetString(OracleService.Filter(json, "Manufacturers[0].Name")));
+            Assert.AreEqual("[50]", Utility.StrictUTF8.GetString(OracleService.Filter(json, "Manufacturers[0].Products[0].Price")));
+            Assert.AreEqual(@"[""Elbow Grease""]", Utility.StrictUTF8.GetString(OracleService.Filter(json, "Manufacturers[1].Products[0].Name")));
+            Assert.AreEqual(@"[{""Name"":""Elbow Grease"",""Price"":99.95}]", Utility.StrictUTF8.GetString(OracleService.Filter(json, "Manufacturers[1].Products[0]")));


should add Filter(json,"") and Filter(json,null)

I'll add it.

neo-modules/src/OracleService/OracleService.cs

Lines 427 to 428 in 0acf93b

if (string.IsNullOrEmpty(filterArgs))

return Utility.StrictUTF8.GetBytes(input);

It returns the unmodified input.

cloud8little · 2021-01-15T03:05:53Z

Smoke test passed.

erikzhang · 2021-01-15T05:03:18Z

Merge?

* init * move * Clean code * Update src/Oracle/Oracle.cs * fix * Update src/Oracle/Oracle.cs * fix oracle * format * fix * optimize * optimize * format * fix * add CheckOracleAccount * fix * Update src/Oracle/Protocol/OracleHttpProtocol.cs * Update src/Oracle/Oracle.cs * fix * format * fix * Update Oracle.cs * fix timeout * fix timeout2 * optimze * format * Apply OracleHttpProtocol.cs * optimize * fix * remove * fix * optimize * Apply recommmendations * fix * add backupTx * sort public key * avoid two txs * send backup sign after timeout * exchange arguments of AddResponseTxSign * format * format * add cache * fix * optimize password * up error code * format * fix and add lock * fix * optimize * up oracle response code * use base64 * fix * remove useless library * fix * optimize * optimize * rename to OracleService * avoid duplicate request * Replace HashsetCache by ConcurrentDictionary * fix * fix * fix * clean code * rename Designate to Designation * Update src/Oracle/OracleService.cs Co-authored-by: Shargon <shargon@gmail.com> * fix * Update projects * Rename * optimize code * static AllowPrivateHost/Timeout * Update src/OracleService/Protocols/OracleHttpProtocol.cs Co-authored-by: Erik Zhang <erik@neo.org> * apply erik's suggestion * fix * fix * fix 2/2 * Update OracleService.cs * CI01118 * add Process * Optimize * Remove unused parameter * readonly * Use System.Threading.Timer * Rename * Update OracleService.cs * Update OracleService.cs * Rename * Add Settings.cs * Move IsInternal to Helper.cs * Update OracleService.cs * Remove Utility alias * Use a single HttpClient * Use HttpCompletionOption.ResponseContentRead * private class * Simplify OracleService.cs * Use Interlocked * fix * Using lock in OnTimer method * Replace lock by concurrentdictionary * Optimize * Simplify * async * Prevent reentry * Optimize * private * Prevent multiple starts * fix * fix * Optimize CheckOracleAccount * Fix OnStop * Reorder methods * Optimize CreateResponseTx * fix * Remove WithTimeout * Add check * add oracle ut * Delete Neo.Plugins.OracleService.Tests.csproj * Delete OracleServiceTest.cs * add try-catch when call Filter * fix and add UT * Remove empty line * Fix OnPersist * Optimize iteration * more optimize * fix ut * fix * Optimize * use random * Json un-indented * Replace nonce for deterministic value * Replace nonce * Add UT for filters Co-authored-by: Shargon <shargon@gmail.com> Co-authored-by: Owen Zhang <38493437+superboyiii@users.noreply.github.com> Co-authored-by: Erik Zhang <erik@neo.org>

init

05f9e74

Tommo-L mentioned this pull request Aug 19, 2020

Asynchronous Oracle neo-project/neo#1738

Merged

Tommo-L and others added 3 commits August 21, 2020 15:53

move

78944c6

Merge branch 'master' into async-oracle

c033672

Clean code

6760302

shargon reviewed Aug 26, 2020

View reviewed changes

neo-modules.sln Outdated Show resolved Hide resolved

shargon reviewed Aug 26, 2020

View reviewed changes

src/Oracle/Oracle.csproj Outdated Show resolved Hide resolved

shargon reviewed Aug 26, 2020

View reviewed changes

src/RpcServer/RpcServer.csproj Outdated Show resolved Hide resolved

shargon reviewed Aug 26, 2020

View reviewed changes

Update src/Oracle/Oracle.cs

76bee8c

Tommo-L added 2 commits September 1, 2020 18:19

fix

68a2f4d

Merge branch 'async-oracle' of https://github.com/Tommo-L/neo-modules …

9101194

…into async-oracle

shargon reviewed Sep 1, 2020

View reviewed changes

src/Oracle/Oracle.cs Outdated Show resolved Hide resolved

shargon and others added 6 commits September 1, 2020 12:40

Update src/Oracle/Oracle.cs

1cd3843

fix oracle

c31654a

Merge branch 'async-oracle' of https://github.com/Tommo-L/neo-modules …

a7b3d5f

…into async-oracle

format

2d53f13

fix

22edae9

Merge branch 'master' into async-oracle

79630a6

Tommo-L marked this pull request as ready for review September 8, 2020 11:03

Tommo-L added 3 commits September 8, 2020 19:11

optimize

2e64c1e

Merge branch 'async-oracle' of https://github.com/Tommo-L/neo-modules …

de131c4

…into async-oracle

optimize

19053d2

superboyiii mentioned this pull request Sep 15, 2020

Preview4 Checklist neo-project/neo#1936

Closed

44 tasks

Tommo-L added 2 commits September 15, 2020 19:59

fix

49abda8

format

66a5497

erikzhang and others added 2 commits January 11, 2021 21:59

more optimize

10e7c2b

fix ut

89a16ba

cloud8little reviewed Jan 14, 2021

View reviewed changes

src/OracleService/OracleService.cs Outdated Show resolved Hide resolved

fix

883f779

Merge branch 'master' into async-oracle

6803733

fix conflicts

e88be60

Optimize

3139d80

erikzhang reviewed Jan 14, 2021

View reviewed changes

src/OracleService/OracleService.cs Show resolved Hide resolved

src/OracleService/OracleService.cs Outdated Show resolved Hide resolved

src/OracleService/OracleService.cs Show resolved Hide resolved

use random

12ef99c

erikzhang reviewed Jan 14, 2021

View reviewed changes

src/OracleService/OracleService.cs Outdated Show resolved Hide resolved

shargon added 4 commits January 14, 2021 15:21

Json un-indented

e51a1a3

Replace nonce for deterministic value

31c0d98

Replace nonce

f1d5e67

Add UT for filters

15e4c49

erikzhang approved these changes Jan 14, 2021

View reviewed changes

shargon approved these changes Jan 14, 2021

View reviewed changes

cloud8little reviewed Jan 15, 2021

View reviewed changes

Merge branch 'master' into async-oracle

c4ec4fe

cloud8little approved these changes Jan 15, 2021

View reviewed changes

Merge branch 'master' into async-oracle

0acf93b

shargon merged commit 2a6ef3a into neo-project:master Jan 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OracleService #326

OracleService #326

Tommo-L commented Aug 18, 2020

shargon Aug 26, 2020

erikzhang Aug 26, 2020

shargon Aug 26, 2020

shargon Aug 26, 2020

erikzhang Aug 26, 2020

shargon Aug 26, 2020

shargon Aug 26, 2020

shargon commented Aug 26, 2020

erikzhang commented Aug 26, 2020

Tommo-L commented Aug 26, 2020

Tommo-L commented Jan 12, 2021

cloud8little commented Jan 14, 2021 •

edited

Loading

Tommo-L commented Jan 14, 2021

erikzhang commented Jan 14, 2021

Tommo-L commented Jan 14, 2021

cloud8little commented Jan 15, 2021

cloud8little Jan 15, 2021

Tommo-L Jan 15, 2021

erikzhang Jan 15, 2021

cloud8little commented Jan 15, 2021

erikzhang commented Jan 15, 2021

	if (string.IsNullOrEmpty(filterArgs))
	return Utility.StrictUTF8.GetBytes(input);

OracleService #326

OracleService #326

Conversation

Tommo-L commented Aug 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shargon commented Aug 26, 2020

erikzhang commented Aug 26, 2020

Tommo-L commented Aug 26, 2020

Tommo-L commented Jan 12, 2021

cloud8little commented Jan 14, 2021 • edited Loading

Tommo-L commented Jan 14, 2021

erikzhang commented Jan 14, 2021

Tommo-L commented Jan 14, 2021

cloud8little commented Jan 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud8little commented Jan 15, 2021

erikzhang commented Jan 15, 2021

cloud8little commented Jan 14, 2021 •

edited

Loading